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Abstract 

The subspace approximation problem Subspace(fc,p) asks for a fc-dimensional linear subspace 
that fits a given set of m points in K." optimally. The error for fitting is a generalization of the 
least squares fit and uses the £p norm of the (£2) distances of the points from the subspace. 
Most of the previous work on subspace approximation has focused on small or constant k and 
p = 1 or 00, using coresets and sampling techniques from computational geometry. 

In this paper, extending another line of work based on convex relaxation and rounding, we 
give a polynomial time algorithm, for any k and any p > 2. This extends a result of Varadarajan, 
Venkatesh, Ye and Zhang [24], who gave an 0{\/log m) approximation for all k and p = 00. The 
approximation guarantee of our algorithm is roughly •\/27p where 7p « yjp/e{l + o(l)) is the 
p*h norm of a standard normal random variable. The approximation ratio improves to 7p in the 
interesting special case when k = n — 1. We also show that the convex relaxation we use has an 
integrality gap (or "rank gap") of 7p(l — e), for any constant e > 0. 

We also study the hardness of approximating this problem. We show that assuming the 
Unique Games Conjecture, the subspace approximation problem is hard to approximate within 
a factor better than 7^(1 — e), for any constant e > 0. The hardness reduction involves a 
dictatorship test which is somewhat different from "long-code" based tests used in reductions 
from Unique Games, and seems better suited for problems of a continuous nature. 
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1 Introduction 



Large data sets that arise in data mining, machine learning, statistics and computational geometry 
problems are naturally modeled as sets of points in a high-dimensional Euclidean space. Even 
though these points live in a high-dimensional space, in practice they are observed to have low 
intrinsic dimension and it is an algorithmic challenge to capture their underlying low-dimensional 
structure. The subspace approximation problem described below generalizes several problems for- 
mulated in this context. 

Subspace(/i;,p): Given points ai,a2, . . . , S I^", an integer k, with < k < n, and p > 1, find 
a fc-dimensional linear subspace that minimizes the sum of p-ih powers of Euclidean distances of 
these points to the subspace, or equivalently, 

m \ Vp 

i=i I 

Note that, here, norm is used as a function of (d(ai, F), (i(a2, V)^ . . . , d{am, V))', the individual 
distances d{ai,V) are the usual £2 distances. 

We describe below the special cases of the subspace approximation problem which have been 
studied previously and the known results about them. 

1. Low-rank matrix approximation problem or the least squares fit (p = 2): Given a 
matrix A G IK'"^"- and < k < n, the matrix approximation problem is to find another matrix 
B € M™^" of rank at most k that minimizes the Frobenius (also known as Hilbert-Schmidt) 

norm of their difference ||^ — = — Bij)^] . Taking the rows of A to be 

points ai,a2, ■ ■ ■ ,am S M", the above problem is equivalent to the problem Subspace(A;,2). 
Elementary linear algebra shows that the optimal subspace is spanned by the top k right 
singular vectors of A, which can be found in time 0(min{mn^, m^n}) using Singular Value 
Decomposition (SVD) [11]. 

2. Computing radii of point sets {p = 00): Given points ai, 02, • • • , am £ I^") their outer (n — 
k)-radius is defined as the minimum, over all fc-dimensional linear subspaces, of the maximum 
Euclidean distance of these points to the subspace (which is equivalent to Subspace(/i;,oo)). 
Gritzmann and Klee initiated the study of this quantity in computational convex geometry 
[12] and gave a polynomial time algorithm for the minimum enclosing ball problem (or the 
problem Subspace(0,oo)). 

(a) For small k: Badoiu, Har-Peled and Indyk [4] gave a (1 -|- e)-approximation algorithm 
running in polynomial time for the minimum enclosing cylinder problem (equivalent 
to Subspace(l,oo)), which was further extended by Har-Peled and Varadarajan [14] to 
Subspace(A:,oo) for constant k. 

(b) For large k: Brieden, Gritzmann and Klee [3] showed that it is NP-hard to approximate 
the width of a point set (equivalent to Subspace(77, — l,oo)) within any constant factor. 
Prom the algorithmic side, the results by Nesterov [19] and Nemirovski, Roos and Tarlaky 
[18] on quadratic optimization imply 0(-y/log m)-approximation for Subspace(n— l,oo) in 
polynomial time. Building on these techniques, Varadarajan, Venkatesh, Ye and Zhang 
[24] gave a polynomial time 0{yJ\og m)-approximation algorithm for Subspace(A;,oo), for 
any k. On the hardness side, they proved that there exists a constant (5 > such that, 
for any < e < 1 and k < n — n^, there is no polynomial time algorithm that gives 
(logm)^-approximation for Subspace(fe,oo) unless NP C DTIME (2P°iyi°sW). 



minimize 

V : dim(V)=fc 
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3. Other values of p: For general p and constant k, a result of Shyamalkumar and Varadarajan 
[21] and subsequent work by Deshpande and Varadarajan [7] gave a (1 + e)-approximation al- 
gorithm with running time O [mn ■ exp{k,p, ^/e)). The running time was recently improved to 
O {mn ■ poly(fe, i/e) + (m + n) • exp(fc, i/e)) by Feldman, Monemizadeh, Sohler and Woodruff 
[10], for the case p = 1. 

For p ^ 2, we do not know any suitable generalization of SVD, and therefore, have no exact 
characterization of the optimal subspace. The approximation techniques used so far to overcome 
this are: (i) coresets and sampling-based techniques: which give nearly optimal approximations but 
only for small or constant k and p. (ii) convex relaxations and rounding: which give somewhat sub- 
optimal approximations mostly for large values of k; the only exception is the result of Varadarajan, 
Venkatesh, Ye and Zhang [24] which works for any k (but only for p = oo). 

Our work 

In this paper, we study the problem Subspace(fc,p) for p < oo, about which little is known in general. 
One motivation for doing so is that often the case p < oo gives significantly better approximation 
guarantees and requires somewhat different techniques to analyze than p = oo. This is evident in 
the work for subspace approximation for small k ([7] and [10] for p < oo versus [4] and [14] for 
p = oo) and in the work on regression ([5] and [6] versus the p = oo case which is solvable by fixed 
dimensional linear programming). Also, in the study of hardness of approximation, the case p = oo 
can often be reduced to a discrete problem; while the case p < oo is inherently of a more continuous 
nature, and requires somewhat different techniques. 

On the algorithmic side, we give a factor jp ■ \J2 — {}/n~k) approximation algorithm for the 
problem Subspace(A;,p) in M", where 7p ~ -sJpjeiX + o(l)) is the p^^ norm of a standard Gaussian. 
Our algorithm is based on a convex relaxation, similar to the semi-definite relaxations used in [18] 
and [24] for p = oo. We give a tighter analysis for general p. We also exhibit gap instance for the 
convex program. We show a gap of factor 7p for Subspace(A;,p) (when k is superconstant) showing 
that our analysis is tight up to the factor of \/2 — {}/n-k). 

We also investigate the hardness of approximation for Subspace(A;,p). We give a reduction from 
the Unique Label Cover problem of Khot [15] to the problem of approximating Subspace(n — l,p) 
within a factor 7p (which can trivially be extended to a reduction to Subspace(/c,p) for k = rf'^^^). 
The reduction is related to the ones used for similar geometric problems in [17], [16] and [2]. 
However, an interesting difference here in comparison to usual reductions is that we use a different 
(real- valued) encoding of the assignment to Unique Label Cover (in terms of the Fourier coefficients 
of the long-code instead of the truth table) which is more natural in our context. This may also be 
useful for other problems of a continuous nature. 

Very recently, our techniques were also extended by Guruswami et al. [13] to give a reduc- 
tion from the Label Cover problem {without assuming the uniqueness property) to approximating 
Subspace(n — l,p) within a factor of 7p. This proves an unconditional NP-hardness for the latter 
problem. 

Other related problems 

Lp-Grothendieck problem. In the k = n — 1 case, subspace approximation problem can be 
rewritten as min||^||^=;^ || AzU^, where the rows of A G represent the points oi, 02, . . . , am and 

z G ffi" represents the unit normal to the subspace we are asked to find. When A is invertible, 
this problem can be shown (using duality in Banach spaces) to be equivalent to a special case 
of the Lp-Grothendieck problem (introduced by Kindler, Naor and Schechtman [17]) which asks 
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for maximizing Mx subject to \\x\\^ < 1. Subspace(ri — l,p) with invertible A, reduces to this 
problem with M = {A~^)'^A~^. 

In this special case, using Grothendieck's inequality and a technique by Alon and Naor [1], one 
can get 0(l)-approximation. Moreover, in this case, the above problem is also equivalent to finding 
diameters of convex bodies given by ||^a;||p < 1 and computing p i— > 2 norm of the matrix A~^. 

/p-regression problem. In the Ip regression problem, we are given an m x n matrix A and a 
vector b G M"*, and the goal is to minimize \\Az — b\\p over all z G M". This is clearly related to 
subspace approximation with k = n — 1, but the fact that z is unconstrained makes it a convex 
optimization problem. Efficient approximation algorithms for the regression problem are given by 
Clarkson [5] for p = 1, Drineas, Mahoney, and Muthukrishnan [8] for p = 2, and Dasgupta et 
al. [6] for p > 1. It is not clear that these results can be employed fruitfully for the subspace 
approximation problem for k = n — 1 where it is required that ||z|| > 1. 



2 Preliminaries and Notation 

Throughout this paper, ||-||p denotes the p-norm. Norms of vectors are taken with respect to the 
counting measure and of functions are taken with respect to the uniform probability measure on 
their domain. When the subscript is unspecified, ||-|| denotes ||-||2- 



2.1 The Subspace Approximation Problem 

We will use a formulation of the problem Subspace(/i;,p) for points ai, . . . ,am, in terms of the 
orthogonal complement of the desired subspace V. Let zi, . . . , Zn-k be an orthonormal basis for 
the orthogonal complement and let Z G denote the matrix with the j'*'* column Z^^^ = Zj. 

Then d(aj, V) = Ha^Zjl^ and the problem of finding (the orthogonal complement of) the subspace 
can be stated as 



minimize 




subject to: p(-')|| > 1 Vj G {1, . . . , n - fc} 

l^Z^h)^Z^h)^ = Vii / is, Z G R"x("-^-) 

For the hardness results we shall be concerned with the special case of the problem with k = n—1. 
For points oi, . . . , G M", let ^ be m x n matrix with A^ = aj. The problem Subspace(n — l,p) 
is then simply to minimize for z G ffi"", subject to \\z\\2 > 1. 

Remark 2.1 It is easy to check that (by a change of variable and suitable modification of A) both 
the norms can be taken to be with respect to an arbitrary measure instead of the counting measure. 
In particular, if A G M"*^", the p-norm is taken with respect to a measure // on [m] and the 2-norm 
with respect a measure on [n], then we change variables to z with Zj = \J v(j)zj and modify Aij 
to Aij{fj,{i)y/P / \fiAj) to get an equivalent problem with norms according to the counting measure. 



2.2 Bernoulli and Gaussian Random Variables 

A Bernoulli random variable is a discrete random variable taking values in {—1, 1} with probability 
1/2 each. A standard normal random variables (or 1-dimensional Gaussian) is a continuous random 
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variable with probability density function l/v27r-exp(— j; /2). We use 7^ to denote the p moment 
of iV(0,l), 



7p 



def 



■ dx 



'2P/2.rfE+i 



27r 



vr 



'^(1 + 0(1)). 

e 



We shall require both upper and lower bounds on moments of a sum of Bernoulli random 
variables by the moment of an appropriate Gaussian. The following upper bound is one direction 
of the Khintchine inequality (see [20]) well-known in functional analysis. 



Claim 2.2 Lei xi, . . . he independent Bernoulli random variables and let ci, . . . ,c/j G M and 
;^ + •••-!- c^. Then for any positive p > 0, 



E 

Xl,...,Xli 



R \ P- 



The following version of the reverse direction, when all Cj's are much smaller than ||c||, can be 
derived using the Berry-Esseen Theorem (as in [23]). A proof of the statement below appears in 
[17] (as Lemma 2.5). 

Claim 2.3 Let x\, . . . ,Xft^ he independent Bernoulli random variables and let ci, . . . ,cji G M be 
such that for all i £ [R], |cj| < r • ||c|| for r S (0, e~^). Then, for any p > 1, 



E 

Xl,...,Xj{ 



E 

i=l 



Ci ' X'l 



> 7M|c||^- l-10T-(log(iA)) 



p/2 



3 Technical Overview 

In this section we describe our results and give a general outline of the sections that follow. 



3.1 Algorithm for Subspace(A;,p) 

We formulate problem Subspace(A;,p) for points ai,a2, ■ ■ ■ ,0,^ S 1^"" in terms of the orthogonal 
complement of the desired subspace V. Let Z € R"'^("~'^') be the matrix whose columns form an 
orthonormal basis for the orthogonal complement of V, then the distance of a point from V can 
be written as d{ai, V) = \\ajz\\^ = (of Z Z'^ aiY^^ . Note that ZZ^ is a positive semidefinite (p.s.d.) 
matrix of rank n — k, all of whose nonzero singular values are 1 and whose singular vectors (the 
columns of Z) specify the (complement of the) subspace V. 

A convex relaxation of Subspace(A;,p) is then obtained by optimizing over arbitrary positive 
semidefinite matrices X and replacing the requirement that the matrix have rank n — A; by a 
condition on the trace of X (see Figure 1). This is similar to the relaxations used in [18, 24]. The 
problem then reduces to giving a "rounding algorithm" which reduces the rank of the matrix X 
(which might be as large as n) to n — k, and achieves a good approximation of the objective value 
of the convex program. 

In keeping with the intuition that the singular vectors of ZZ-^ span the orthogonal complement 
of V, our algorithm looks at the singular vectors of the matrix X obtained by solving the convex 
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relaxation. It then divides the singular vectors into n — k "bins", and constructs one vector for 
each bin by taking a random linear combination of vectors within each bin. 

Our algorithm described in Section 4 achieves an approximation ratio of 7^ • (2 — for 
Subspace(/i;,p), where q = 2 ■ \p/2]. (See Theorem 4.4.) 

We remark that the problem of obtaining low-rank solutions to a semidefinite program was also 
considered by [22], and was addressed by simply taking random (chosen according to a Gaussian) 
linear combinations of the singular vectors of the relevant matrix. However, in their case, they 
were' only interested in satisfying the constraints, with an error depending inversely on the rank 
parameter. In our case, we require a rank n — k positive semidefinite matrix, all of whose eigenvalues 
are exactly 1. Since the only constraint enforcing this is a constraint on the trace of the matrix, 
even a small multiplicative error in satisfying the constraint can make some singular values quite 
small. To resolve this, we proceed by dividing the singular vectors in various bins and take Bernoulli 
linear combinations, do directly generate the orthogonal singular vectors. 

3.2 A gap instance 

In Section 5, we show that the convex relaxation we use has an integrality gap, or more correctly 
"rank gap", of 7p(l — e), for any constant e > 0. Given any constant e > 0, we construct points 
bi,b2, ■ ■ ■ ,bm S such that the optimum for Subspace(n — l,p) on these points (a rank 1 p.s.d. 
matrix) and the optimum for its corresponding convex relaxation (a rank n p.s.d. matrix) are at 
least a factor of 7p(l — e) apart. We first show such a gap for the continuous analog of Subspace(n — 
l,p) where the point set is the entire M" equipped with Gaussian measure (Theorem 5.1). We then 
discretize this example to get our final integrality gap construction (Theorem 5.2). 

This also gives a gap of factor 7^(1 — e) for Subspace(A;,p) for any super-constant k = k{n), since 
an instance of Subspace(n — l,p) in M"" can be trivially converted (by adding extra zero coordinates) 
to an instance of Subspace(A;,p) in with k{n') = n — 1. 

3.3 Unique-Games hardness 

In Section 6, we describe a reduction from Unique Label Cover to the problem of approximating 
Subspace(n — l,p) within a factor better than 7p (for a constant p > 1). By a trivial reduction from 
Subspace(n — l,p) to Subspace(fe,p) for any k = n^^^\ this gives the hardness of approximating 
Subspace(fc,p) better than 7p, assuming the Unique Games Conjecture. 

To understand the intuition for the reduction, let us consider the simpler problem of testing 
whether a given function / : {-1, 1}^ ^ {-1, 1} is a "dictator" i.e. /(xi, . . . ,xr) = xi for some 
i G [i?], which is a useful primitive in such reductions. The problem is to design an instance X of 
Subspace(re — l,p) and interpret the description of / as a solution to T. The required property is 
that if / is a dictator then the corresponding subspace fits the points in X with small error. On the 
other hand, if / is "far from being a dictator", the error is required to be larger by a factor of 7^. 

In most reductions, / is assumed to be described by its truth table. However, if we want to 
interpret the input simply as the coordinates of a vector z, there is no way to enforce that the 
coordinates be boolean. It turns out to be more convenient if we require / as a list of its Fourier 
coefficients which can be thought of as a vector with arbitrary real numbers coordinates and norm 
1 (since E[/^] = !)• Also, since we are only interested in dictator functions, it is sufficient to ask 
for the "level 1" Fourier coefficients /({I}), . . . , f{{R]). 

In particular, consider the input being described by R real numbers hi, ... ,bf( such that J2i bf = 
1 and we think of it as describing the function fbixi, . . . , xr) = 61 • xi + • • • + 6^ • x/j. We also think 
of bi, . . . ,bfj as the normal to some R — 1 dimensional subspace of M^. Let the points be given 
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by • X for each vector x S {—1, 1}^, so that the objective of the subspace approximation 

problem is exactly If / is a dictator, i.e., one of the 6j's is 1 and others 0, then = 1. 

Also, if it is far from a dictator in the sense that maxj hi < t for a small constant r, then ~ 7p 
by Claim 2.3. 

Translating this intuition to a reduction from Unique Label Cover turns out to be slightly techni- 
cal due to the fact that we need to consider one function for each vertex of Unique Label Cover and 
all bounds on norms do not hold for individual functions but only on average. Similar technicalities 
arise when working with the ip norm in [17] (though they specify functions by their truth tables). 



4 Approximation Algorithm via Convex Programming 



To relax the minimization problem for Subspace(/c,p) to a convex problem, we rewrite the distances 
||a^Z|| in the objective as {aj Z Z'^ aiY^^ . Noting that ZZ^ is a positive semidefinite matrix of rank 
n — k, we get the following natural relaxation similar to the one used in [18, 24]. 



Minimization Problem 



minimize I \aj ZZ 




subject to: P^^^|| > 1 Vj G {1, . . . , n - fc} 



Convex Relaxation 



minimize I \ afXa, 



ip/2 



i/p 



Ki=l 



subject to: Tr(X) > n — k 

X £ R"'''" 



Figure 1: The problem Subspace(/c,p) and its convex relaxation 

Note that this relaxation removes the constraint on the rank and relaxes the constraint on the 
length of the individual vectors Z^^^ to the trace of entire matrix X. Also, the objective function 



is written as \ aJXa. 



|P/2 
|P/2 



1/p 



which is not convex. However, for solving the convex program, we 



can work with - ^aJXai^"'", which is convex for p>2. 

In Figure 2, we give a "rounding algorithm" for the relaxation. Note that the problem here is 
not really to round the solution to an integer solution as with most convex relaxations, but instead 
to reduce the rank of the solution to the program, while obtaining a good approximation of the 
objective. 

We shall show the algorithm outputs a matrix Z of rank n — k which achieves an approximation 
ratio of 7p • y^2 — i/n-fc in expectation, for even integers p > 2. An approximation guarantee for 
other values of p can be obtained via Jensen's inequality. We state the dependence on n — k precisely 
as we shall be interested in the case n — k = 1. For notational convenience, we shall use On.fc to 
denote the quantity \/2 — ^/n-k in the rest of this section. 

It is clear that the columns of the matrix Z given by the algorithm form an orthonormal set 
since they are all in the span of distinct eigenvectors of X, and are normalized to have length 
1. However, this assumes that the lengths of the vectors yj are nonzero. Since a vector yj is a 
weighted sum of orthogonal vectors, \\yj\\'^ = "^t&s- ^i- '^^^ following claim gives a lower bound on 
this quantity which is also useful in bounding the approximation ratio. 
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Input: A matrix X £ M"^'" satisfying / ^ X ^ and Tr{X) >n-k. 

1. Express X in terms of its singular vectors as X = Ylt=i ''^txtxj where the vectors 
xi, . . . ,Xr form an orthonormal set and Ai > A2 > • • • > Aj. > 0. 

2. Partition [r] into n — k subsets Si, ... , Sn-k- Start with Si = ■ ■ ■ = Sn-k = Then 
for t from 1 to r do: 

(a) Find the set Sj for which X^^/g^, \t' is minimum. 

(b) Set Sj := SjVJ{t}. 

3. Pick r independent Bernouhi variables bi, . . . ,br £r {—1, !}• For each j £ [n — k], let 

4. Output the matrix Z G M"x with Z^^) =^ -f^ 



Figure 2: The rank reduction algorithm 

Claim 4.1 Lei Si, ... , Sn-k be the partition constructed by the algorithm in step 2. Then 

ViG[n-fc], VA* > ^ 



/ J <- — 1 



n,k 

Proof: Let jo =^ argmin^- |l]te5j -^*} =^ ^*g5jo ^t- ^ =^ i^o} U {j \ \Sj\ > 1}. 

Note that the algorithm ensures that \Sj\ > for all j but in T we discard the singleton sets. We 
will show that s* > 1/ (2 — VlQl); which will prove the claim since \Q\ < n — k. 

We argue that for each j £ Q, j ^ jo, "^tes '^t — '^^*- *^i^' *i maximal index 

in Sj. At step t = tj, tj was added to set Sj and not to the set Sj^. Hence, 

J2 xt < E ^* ^ 

Also, there exists at least one to £ Sj^ such that to < tj- This is because Sj was non-empty at step 
tj (otherwise it would be a singleton). But then Xtj < Xt^ < s* and, hence, '^teSj -^^ — ^**- 

Finally, we note that for each j ^ Q, Sj contains exactly one element t, the eigenvalue Xt 
corresponding to which is at most 1. Thus, 

(|Q|-l)-2s* + s* + (n-A:-|Q|)-l > ^ At > n-k, 

te[r] 

which completes the proof. ■ 

The following lemma proves the required approximation guarantee for the expected p*^ moment 
of the distance a single point from the orthogonal complement of the column span of Z. 

Lemma 4.2 Let X be the solution of the convex relaxation and let Z be the matrix returned by the 
algorithm. Also, let p be even. Then, for each i G [m] 

, p/2 



E 
z 



< 
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Proof: We can expand ||af Z||, using Wj to denote (|aj, Z^^'^^, as 







' (n-k \ 




" (n-k \ 


11 T ryUP 


= E 
z 




= E 















E 
z 



Note that the Wj-s are independent random variables since each Wj only depends on ht such that 
t G S'j, and the sets are disjoint. Using the multinomial expansion and the fact that p is even, the 
above can be written as 



E 



' n~k 



p/2 



E 



p/2 

Pl,... ,Pn~k 



E 



2P, 



pi,...,p„-k 




2p, 



The following claim then finishes the proof. 



Claim 4.3 E 



< 7?- 



Proof: The proof follows an application of upper bound on a sum Bernoulli variables derived in 



Claim 2.2. We expand E[Wj^'] as 



E 



E 



2pj 



Claim 2.2 gives that E 



Et65, bfV^fXt 



Et65, bt ■ 



E 



2p, 



Ete5, -^^ 



2p, 



that 72pj < 7p (since 2pj < p) proves the claim. 



< 72^ • ( Etes, (oi, xj)^ )■ ' and noting 



For each j, let denote Etes {o-i, xt)'^ and let Aj denote Eies Af. Using the above claim 
we get that 



E 
z 



I T ry\\P 



< 



E 



Plv,Pn-fc 



p/2 

Pl, • • • ,Pn-k 



p/2 



. ryP 
Ip- 



Claim 4.1 gives that i/a^ < a^^,. Also, we have that Dj = ^^ \t{ai,xt)'^ = aJXai. 
Combining these gives Ez ■^||2] — 7p " '^n fc ' {'^J ■^'^iY^^ which proves the lemma. ■ 

An approximation guarantee for other values of p can be obtained via a standard application 
of Jensen's Inequality. We state the dependence on n — precisely as we shall be interested in the 
case n — A; = 1 in the later sections. Notice that the approximation factor is 7g, where q = 2 • [p/2] , 
in the case n — k = 1 , and thus matches the integrality gap and unique-games hardness that appear 
in the later sections. 
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Theorem 4.4 Let X be the solution of the convex relaxation and let Z be the matrix returned by 
the algorithm. Let p > 1 and let q = 2 ■ [p/2] be the smallest even integer such that q > p. Then, 



E 
z 



p/2 



Proof: (Proof of Theorem 4.4) By the concavity of the function f{u) = ul"^ and Jensen's 
Inequahty we have that 



E 
z 



( m \ Vp 




m 


(Eii»r^E) _ 


^ (1 








.i=l 



Vp 



and by hnearity it suffices to consider a single term of the summation. Another apphcation of 
Jensen's (using p < q) and Lemma 4.2 give that 



E 
z 



E 
z 



I T ry\\<l 



p/q 



< 



E 

Z L 



I 2" '711'^^ 

|aj Z.1I2 



p/q 



p/2 



which completes the proof of the theorem. 



Remark 4.5 Our results are stated in terms of the expected approximation ratio achieved by the 
algorithm. However, one can get arbitrarily close to this ratio with high probability, simply by 
considering few independent runs of the algorithm and picking the best solution. In particular, one 
can achieve an approximation guarantee (1 + e) • 7^ • \/2 — (}/n~k) with probability 1 — Pe, by using 
0(1A • log(Vpe)) runs. 



5 A Gap Instance for the Convex Relaxation 

Here we describe an instance of Subspace(n — l,p) such that the value of any valid solution (which 
is of rank 1) is at least 7p times the value of the convex relaxation. Note that approximation ratio 
of the algorithm for the case n — k = 1 (and even p) is exactly jp and hence this shows that our 
analysis is optimal for this case. 

This also gives a gap of factor 7p for Subspace(A:,p) for any super-constant k = k{n), since an 
instance of Subspace(n — l,p) in can be trivially converted (by adding extra zero coordinates) 
to an instance of Subspace(A;,p) in M" with fc(n') = n — 1. 



5.1 A continuous gap instance 

Recall that an instance of Subspace(n — l,p) can be expressed as min|j2||^=i II^^H^ for A G M'"^"*, 
where 01,02, ■■ ■ ,Om form the rows of A. We consider a continuous generalization of this, where 
instead of points, we are given a probability distribution on M" with density function /i(-), and 
objective is: 

Vp 

mm I / \{a, z)\^ fi{a)da 

lkll2=l \Ja& 

The corresponding convex relaxation is 



min ( / (a^ XaY^^ ii(a)da] 



Vp 



I)pX)pO 

Mx)=i 
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We first show that Gaussian measure on M", i.e., i.i.d. coordinates from A^(0, 1), gives a gap 
instance for the above problem. 

Theorem 5.1 Given rj > 0, there exists ng € Z such that for all n > hq if fj, is the Gaussian 
density function on M" with each coordinate having mean and variance 1, then 



mm 

\\A\2 = 



( / \(a, z)f n(a)da] > 70(1 — 17)- min ( / (a^ Xa) n(a)da] 



Tr(X)=l 



Proof: We first consider the value of the LHS. By the rotational invariance of the Gaussian 
measure, the value is equal for all z and we can restrict ourselves to z = ei. 

min I / \{a, z)\^ ij,{a)da\ = I / \{a,ei)f fi{a)da\ 



/ I'^iT xrf/o dai ■ da2 ■ ■ ■ ■ da. 



(27r)^/2 



7p- 



In comparison, the optimum of the convex relaxation can be upper bounded by using the matrix 
min I / [a^ Xa\ n(a)da] < —;= { / llalP - — —rdai-da2----dan\ 

Tr(A)=l ^ ' 

_ , , , r^- — r-rr^ dr ■ duo \ 



00 p-^^h 

r^'+P-^-^dr- / dtJ 



"li \ (27r)("-^V2 27^ r(V2) J 



n y r("/2) / V " 

2 

where the third equality used that f^^§n-i duj = area(§"~-^) = and ^n+p-i '^^^ dr = 

7n+p-i/2- Choosing n ^ p/r] then proves the claim. H 

5.2 Discretizing the gap example 

A discrete analog of the above, i.e., picking sufficiently many samples from the same distribution, 
gives us our final integrality gap (or "rank gap" ) example. 
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Theorem 5.2 Given any rj > 0, there exist mojUQ G Z such that for all m > ttiq and n > no, if 
we pick i.i.d. random points ai, a2, ■ ■ ■ , am G IK" with each point having i.i.d. N{0, 1) coordinates, 
then with some non-zero probability, 

In other words, there exist points bi,b2, ■ ■ ■ ,bm G IR", where bi '= m~^/^ ■ Oi, giving the desired 
integrality gap example. 

The theorem can be proved by using the continuous gap instance, and concentration bounds 
for the samples ai, . . . , Om- We defer a full proof to the appendix. 

6 Unique- Games Hardness 

6.1 Khot's Unique Games Conjecture 

We shall show a reduction to subspace approximation problem from the Unique Label Cover prob- 
lem defined below. 

Definition 6.1 An instance of Unique Label Cover with alphabet size R is specified as a bipartite 
graph U = (F, W,E) with a set of permutations {tt^w '■ — ^ W\}{v,w)^E- ^ labeling C : V UW ^ 
[R] is said to satisfy an edge {v,w) if C{w) = 'Kvw[C{v)). We denote by val(ZY) the maximum 
fraction of edges satisfied by any labeling C 

The Unique Games Conjecture proposed by Khot in [15] conjectures the hardness of distinguishing 
between the cases when the optimum to the above problem is very close to 1 and when it is very close 
to 0. This conjecture is an important complexity assumption as several approximation problems 
have been shown to be at least as hard as deciding if a given instance lA of Unique Label Cover 
problem has val(^) > 1 — e or val(ZY) < 5 for appropriate positive constants e and 6. 

Conjecture 6.2 (Khot [15]) Given any constants e, 5 > 0, there is an integer R such that it is 
NP-hard to decide if for given an instance U = {y,W,E) of Unique Label Cover with alphabet size 
R, val(ZY) > 1 - e or val(Z^) < 5. 

6.2 Reduction from Unique Label Cover 

We will now prove Unique-Games hardness of approximating Subspace(?i — l,p) within a factor 
better than 7^. As in Section 5, this also gives a hardness approximating Subspace(A:,p) for k which 
is a sufficiently large function of k, by a trivial embedding of the given instance R" into such 
k{n') = n — 1. If we want n' to be a polynomial in n, this will give a hardness for all k = n^^^\ 

We describe below the reduction from an instance U = {V,W,E) of Unique Label Cover with 
alphabet size R to Subspace(?i — l,p). The variables in our reduction will be of the form byo,i for 
each w G W and i € [R]. We denote the vector {bw,i, • • • , 6io,_r) by and for each v £ V, define 
b. =E»e N{v)['^wv0^w)]- For any b G M^, we define the function : { — 1, 1}^ — ^ M as 

R 

fb{xi,...,XR) = ^Xi-bi 

i=l 
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Norms for functions are defined as usual (over the uniform probability measure). Note that \\fb\^ = 

1 1 1 1 2 

||b||2. When the exponent in the norm is unspecified, ||-|| denotes ||-||2- 

Given an instance li = {y^W,E) of Unique Label Cover we output the following instance of 
subspace approximation, for a suitable constant B to be determined later: 



minimize 



subject to: 



E 

{v,w)eE L 

E 

{v,w)eE 



fbv 1 1 p 



\\fbA2 



+ B- E 










E 


11, ||2" 
II "w II 2 


> 1 









Note that the variables in the problem are only the vectors b^^ for all w £ W. It is easy to verify 
the functions fb^ and fb^ can be generated by application of an appropriate operator A. In the 
proof below we shall often drop the subscript on the permutations tt^v when it is clear from the 
context. Note that value of instance of Subspace(n — l,p) is actually the p*'* root of the above 
objective. Let (opt)^ denote the optimal value for the above objective (so that opt is the optimal 
value for Subspace(n — l,p)). 



Completeness 

The following claim shows that the optimum of the subspace approximation problem is low when 
the Unique Label Cover is instance is highly satisfiable. 

Claim 6.3 If ya\{U) >l-e, then (opt)^ < 1 + e • B • 2^. 

Proof: By assumption, there exists a labeling C : V UW ^ [B\ such that ¥[v,w)<^E[^i^^) 7^ 
■Kwv{C{w)y\ < e. We construct a solution the above instance of the subspace approximation problem, 

taking hw,i = 1 if C{w) = i and otherwise. It is easy to check that 'E,(v,w)&E \\fbu,\\2 ~ ^• 

We now bound the value of the objective function. First note that fb^ = '^w<^N{v)[f-K(bw)\ is 
bounded between -1 and 1, which implies 'K[v,w)<^E 
can use Jensen's Inequality to get 



E 

(v,w)<^E L 



I /f)u / TTmi) (few ) 



E 



E 



< 1. To bound the second term, we 



< E 



Note that 
Hence, 



fn^^^ib^^) ~ f-^^^b^'^) " equals 2^"^ if 7r^,^(£(i(Ji)) / 'nw2v{C-{w2)) and otherwise. 



E 



2P- 



V,Wl ,W2 



(^(^2))] 



< 2P-M p [tt^.Mwi)) £{v)] + P [7r^2vmw2)) ^ Civ)] 

\y,wi v,W2 

< 2P-e. 



Combining the two bounds above gives (opt)^ < 1 + e • S • 2^. 
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Soundness 



For the soundness, we need to prove that if val(ZY) < 5, then opt > 7p • (1 — z^) where is a smaU 
constant dependmg on e and 6. We first make some simple observations about the optimal solution. 



Claim 6.4 For any optimal solution {b^j^g^y to the above instance of Subspace(n — l,p), it must 
be true that 



1. ¥.(v,w)&E 



lb,: 



1 



Proof: Since scaling all vectors by a constant less than 1 can only improve the value of the ob- 

1. Then 



jective, we can assume that for the vectors {b^j^giy in the solution '&[v,w)&E 
Jensen's inequality gives 



E 



E 

(v,w)<^E 



E [7r.„'i,(b^')] 



< E 

{v,w)<^E 



1. 



To deduce the second fact, we show that there exists a feasible solution {bit,}^„gvi/ such that 
opt < 7p- For all w € W , we take = (Vv^, • • • , V^)- The solution is feasible since ||bu,|| = 1 



for each w £ W and also ]E(v,w)eE 
Bernoulli variables and lib,, 1 1 = 1, C 



\\fbv /7r(6u,)| 



0. Also, since fb^ is a linear function of 



aim 2.2 gives that for each v gV, ||/fe„|L < 7p. 



We show that if val(ZY) < 6, then in fact the first term itself is approximately 7p. As is standard 
in Unique Games based reductions, the proof proceeds by arguing separately about the "high- 
influence" and "low-influence" cases. However, since the inputs for our problem are not in the form 
of a long-code but the vectors b, we will use maxjg/j{|6j| / ||b||} as a substitute for influence of the 
j^th variable on the function /;,. 

For the vertices v € V where the functions have no influential coordinates, the Central 
Limit Theorem shows that ||/fe„||p is very close to jp. We then show that the contribution of the 
remaining vertices to the objective function is small. 

Below, we define Si to be the set of vertices corresponding to low influence functions and 
divide the remaining vertices into three cases which we shall analyze separately. The parameters 
r, /3 G (0, 1/2) will be chosen later. 



Si 

S2 



Sa 



dcf 



def 



dcf 



dcf 



V£V\S2 



max||6^, ill < T • ||bt; 



lb,: 



<(l-/3) 



E 



|b„ 



weN{v) L J 
3i s.t. \bv^i\ > T- ||b„|| and 



weN{v) 



> V4-||b^||] < V4 



y\(5iuS2uS3). 



Since (xi, . . . , xr) = 6„^i ■ xi + - ■ ■ + b^^R ■ xris a linear function of Bernoulli variables, Claim 
2.3 gives that 



V^;g5i ||/,J|^ > 7M|b„||^ l-10T-(log(iA)) 



(1) 
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Note that the norm H/fej^Hp may be unbounded for individual vertices. Hence we will use the 

quantity K(v,w)€E '^{S^ji'^) ' W^vW'^ as a measure of the contribution of the set Si to the objective, 
where is the indicator function of the set Si. Claims 6.5, 6.6 and 6.7 help bound the 

contribution of the sets 5*2, S3 and S4. 



Claim 6.5 



E 

{v,w)eE 



(l-l|5,}W)-||b. 



>l-/3 



Proof: Since = lKweN{v)['^wv{i>bu,)], being in ^2 means that on average, many vectors 
differ from b^. We use this to get a bound on the measure of 52. We have 



|b,r<(l-/3)- E 



as ||b^|| = ||7r^i,(bt„)|| and that is the mean of ttwvO^w)- Now, since ||b|| = \\fb\\, we get that 



^ /?• E 


"lIU l|2" 

l|bu,|| 


< E 


l|bu.|| 


- Ilb.f 


■w£N{v) 




w£N{v) 






^ /?• E 


llb^f" 


< E 


||'^«)i)(buj) b,)|| 


weN(v) 











/?• E l{s,}{v)-\\K 
{v,w)eE L ^ ' 



< E 
{v,w)eE 

< E 
(v,w)eE 



\fhv fn{bw) lip 



2/p 



< E 
\{v,w)eE 



\fbv /7r(fe^) 



(since < ll/L) 



(using Jensen's Inequality) 



where we used the assumption that E[v^w)<^E ||/6„ ~ /7r(bti,)||^ ^ Tp/b. This gives that 



E 

{v,w)&E L 



(l-l|5,}(^^))-||b„f >(l-/3)- E {l-t{s,}{v))-\\K 

J (v,w)&E L 



>(l-/3). 1 



>!-/? 



|b,. 



7^ 



1 from claim 6.4. 



The second inequality above used that E(t,,i„)g_B 

Claim 6.6 E(,,^)ei=; \l{s,}{v) ■ < 16/-' • -^I/b^/". 

Proof: Consider a vertex v ^ S^. Since we know that v ^ S2, we get that 



w&N{v) 



[||b^|| >2||b,||] < 



EweN{v) 



|b„ 



4||bJ 



< 



4-4/3' 



14 



Fix and i G [R] such that > r • ||b^|| and FweN{v) [\K,7T^^{i)\ > • l|bu.||] < V^- By 

union bound, 



weN(v) 



|b^|| < 2||b„|| and < • l|b^ll] > 1-^^-1/(4-4/3) > i/^ 



Using this we can again say that ||bt, — 7r^„?;(b^„)|| must be large on average and, hence, derive 
a bound on the measure of 53. 



E 



|bt) 7r^^(b^ 



> E 

w£N{v) 



\bv,i ^u),7r(j) I 



> y4-\T-\\K,\\-r/4-2\\h^\ 

> -7l6-||b^f. 



As in the previous claim, we use this to conclude that 

^ E [l{53}W-||b.f] < 16/^- IE J||A„-/.(mII2 

{v,w)&E L -I {v,w)&E L 



\ (v,w)eE 



I fbv fn{bn,) I 



2/p 



Claim 6.7 E(.,n,)eE [Msi}i^)] < '^V-'- 
Proof: Since v ^ SiL) S2D S3, we know that 

3i G [R] such that P [|^«;,^„„{i) | > • l|b.a,||] > V4- 

wGN{v) 

Construct a labeling for Vl by assigning to each v £ V, the special label i as above, and to each 
w G W, a random label j satisfying \hu,j\ > f/4 • ||b^„||. For, w £ W when no such j exists or for 
V ^ S'4, we fix a label arbitrarily. 

Note that there can be at most ^^t^ choices of j satisfying \bwj\ > '^/4 - ||b^||. By the condition 
on i, we know that, in expectation, the labeling satisfies 1/4 • '^^/le fraction of the edges incident on 
a w G S'4. Since the fraction of edges satisfied overall is at most 5, we get that 



E [l{s,}{v)-^V(i4]<5 

{v,w)£E 



E [1{S,}{V)] < 64Vr2- 

{v,w)eE 



Let ly denote lOr • (log(VT))''^^. Using these estimates, we can now prove the soundness of the 
reduction. 

Lemma 6.8 Ifva\{U) < 5, then for the reduction with parameters B,t and /? = 



(opt)P > . 1 



^_PT^ _ Wp ■ Yp _ P% (646 



r2 B^P 



2 V 



15 



Proof: Using (1) we have that 



p/2 



(opt)P > E [l|5,}W-7r(l-^)l|b.||I] >ll-{l 



\{v,w)eE 



We lower bound l^Si}{v) by 1 — 1{5'2}(?;) — t{s-i}{v) — t^Sijiv)- Claims 6.5 and 6.6 and give 
bounds on the first two terms (with (3 = r^). 



E 

{v,w)eE 



{1 - l{s,}{v)) ■ \\K\ 



E 

{v,w)eE 



1{S3} lib,, I 



> 1 - r 



2 '^'^P 



r2 B^P ' 



r2 _g2/p 



We bound the third term using Claim 6.7 and Holder's inequality 



E 

{v,w)eE 



1{54} l|bt 



y{i','!«)6£; 

/ 645 \ 2 
< 1^1 -Tp, 



{p-2)/p . . 2/p 

/ \iv,w)eE J 



where the last bound used that since opt < 7p (see Claim 6.4), we must have 



E [||b„||f] 

{v,w)eE 



{v,w)&E (v,w)eE L 



Combining the bounds for the above three terms proves the lemma. 

For a small constant rj such that 7?(log(Yr;))''/^ < '^^''^'^/^o, choosing parameters as 



def 2 / 
T = TJ /p, 



del ( i! 

' 64' 



def / 40p7^ \ 

B = ^ , and e 

\ 7? • rM 



\PlpJ 
def r] 



2P • B 



in Lemma 6.8 would imply that opt < 1 + in the completeness case and opt > 7p • (1 — fy) in the 
soundness case. This gives the following theorem. 

Theorem 6.9 For any p > 2 and sufficiently small constant t], there exist constants £,6 > and a 
reduction from Unique Label Cover to Subspacefn — l,p) such that i/val(ZY) is the fraction of edges 
satisfiable in the given instance of Unique Label Cover and opt is the optimum of the instance of 
Subspacefn — l,p), then 

ya\{U) > 1 — e =^ opt < 1 + r? and 
val(ZY) <6 opt > 7p • (1 - 7]). 
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A Proof of Theorem 5.2 

We restate the theorem below. 

Theorem A.l Civen any 77 > 0, there exist mo, no G Z such that for all m > thq and n > uq, if 
we pick i.i.d. random points ai, 02, . . . , Om G K" with each point having i.i.d. N(0, 1) coordinates, 
then with some non-zero probability, 

^ '—1 ' Tr(X)=l 

In other words, there exist points bi,b2, ■ ■ ■ ,bm G M", where bi '= m~^/^ • Oi, giving the desired 
integrality gap example. 
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Proof: Let ai,a2, ■ ■ ■ ,am be i.i.d. random points in R", where each point has i.i.d. A^(0, 1) 
coordinates. Then, as we have seen above 



for y G S'^-i, 
for y G S"-^ 



E[\{a,,y)n = / \{a,y)\Pf^ia)da = ^P, 
Var [| {a,,y)\P] = E [| {a,,y)\'^] - E [| {a,,y) = 7^ - 7'/, 
By Chebyshev's Inequahty, 



i=l 



< 



I2p 

9 2p 



Let M be any 5-net of the unit sphere (i.e., M C S"""*^ such that for any z G S"^"*^, there exists some 
y G M such that \\y — z\\2 < 6), where (5 is a parameter that wih be picked later. It is known (e.g. 
see Claim 2.9 in [9]) how to construct such (5-nets of S""^ with size as small as \J\f\ < (f )""• 
using union bound over M 



1 



-5]|(a.,y)r>(l-e)7^, forallyGAA 



4 = 1 



^ . (f )" • (7g - Ip') ^ 3 

r2o,2P 4' 



as long as we choose m large enough so that 

4 • (I)" • (7| - I'p') 



m > 



e^-fp^ 



For any z G , using y £ Af closest to it 



'^\{ai,z)\P = ^ |(ai,y) + {ai, z - y)\P 
1=1 

m m 

> ^ lloillf"-^ . 



1=1 



1=1 



i=l 



Therefore, 



min - I (a„ z)^ > (1 - e)7^ - ^ ||a.||r ' 
^ i=i «=i 



> 



But we also know that 
E 

Var 



a,; 



ip-l 



;u(a)rfa = n(P-i)/2(l + o(l)) 



•ill 2 



E 



a,; 



|2p-2 



'«II2 



E 



«II2 



n(P-i)(l + o(l)) 



By Chebyshev's Inequality, 

-E 



iP-i 



112 



> (l + e)n(f-i)/2(;^^^(^)) 



j=i 



< 



1 + 0(1) 



Now 
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Hence, choosing m > ^/e^, we have 

" ^ m 

- Yl ll«^lir' ^ (1 + + o{l)) 

Ul . 

1=1 

Putting these together, 

^ m 

min ±y\{a„z)\P>{l-ehP-p6{l + e)n^P-'^/\l + o{l)) 
|2|L=i m ^ ^ 



i=l 



> 3/4. 



OveraU, choosing 



def 1] dcf 

e = — , = 



2 P 



- , and m > max 



we get 



(8 + jy2)pn(P-i)/2 

-I ™ / 2 \ 



' 9 f ' 



mm 

11^112 = 



'P 



1 

>2- 



On the other hand to analyze the value of the corresponding convex relaxation, we use 

°]= / \\ar,fi{a)da = n'/%l + oil)) 

E[||a,||^]^=n^'(l + o(l)) 



E[||a,||^] 



Var[||aJ|Pl =E 



^« II2 



< 



Again by Chebyshev's Inequality, 

" ^ m 

-Ell«^ll2>(l + V2)n^/'(1 + 0(1)) 

Choosing m > s/??^, we get 

^ m 

-Ell«^ll2<(l+V2)n''/^(l + o(l)) 
Therefore, the convex relaxation satisfies 

^ m. 

-5^|Vn,-af/a,f/'<(l + V2)(l + o(l)) 



4(1 + 0(1)) 



i=l 



4(1 + 0(1)) ^ 1^ 
~ mry2 2 



1 

>2- 



Hence, 



mm 

I|2ll2 = 



-VKG„z)r >(l-r?).7p. min -LVlafXa^f^M 

\ i=l / Tr(X)=l \ «=1 / 



> p 

> 0. 



min -j;Ka„z)r>(l-.74).7^ and -Y^\i/n ■ aj la^f < {I + v/2) {I + o{l)) 



ll^ll 



i=l 



i=l 
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B NP-hardness of Subspace Approximation 

In this section, we show unconditionahy that the problem Subspace(n — l,p) is NP-hard, for p > 
2, using a reduction from the Min-Uncut problem on graphs. Such a result was also obtained 
independently by Gibson and Xiao (personal communication). 

Min-Uncut problem: Given a graph G = (V, E), find a bipartition of its vertices V = SUT that 
minimizes the number of edges with both endpoints on the same side of the bipartition. 

Let |y| = n and \E\ = m. Min-Uncut problem is known to be NP-hard, i.e., for some 1 < t < m 
it is NP-hard to find if the Min-Uncut has at most t edges. We give a polynomial time reduction 
from Min-Uncut to subspace approximation as follows: Given an instance of Min-Uncut, construct 
a matrix A G m(™+«)x" such that 



where N is an integer polynomially large in n and m which will be chosen later. 

Yes case: The Min-Uncut has at most t edges. Define Xi = 1 if i £ S and Xi = 1 if i £ T . Using 

this X G {-1, 1}" we get OPT < \\Ax\\p = t2P + Nn. 

No case: Otherwise, for any bipartition the Min-Uncut has at least i -|- 1 edges, i.e., for any 
X G { — 1, 1}" we have X]jj6-B(^« ~^ ^j)^ ^ + 1)2^- Now divide the sphere of radius ^/n into two 
parts as follows: 



n 




S = {y : \\y\\2 = Vn and \yj\ G (1 — e, 1 + e) for all j £ [n]} 
T = {y ■■ \\y\\2 = Vn and y ^ S}, 
where e < Yp • (m + 1). For any y £ T, 

• Case 1: \yi\ = 1 + Si > 1 + e for some i. Then, 




• Case 2: \yi\ = 1 — £i < 1 — e for some i. Then, 



= n- (1-e-). 



Hence, there exists some k such that 




Therefore, using the same analysis as in the previous case, we get 
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Using the above property of y E T, we get 

n p 

ijeE j=i j=i 

> Nn+ ^ 



For any y £ S, 



16n^ 

> + Nn using N > 2^^'^r?m{m 



E + + NY.y''j>{^- + l)2P + iVn 

jjeB i=i 

> (1 + 1)2^ + iVn 

1 



> t2P + iVn using e < , 

p{t + 1) 
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